System and method for processing a video signal with reduced latency

ABSTRACT

Disclosed is a video processor for a magnifier camera. In particular, the disclosure relates to a video processor that eliminates the use of a frame buffer. This, in turn, reduces the latency otherwise present in the video signal. The disclosed video processor also allows selected portions of the display to be shaded. This highlights the non-shaded portions of the display while at the same time allowing the entire object to be perceived by the user.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, and claims priority toapplication Ser. No. 15/895,211, filed on Feb. 13, 2018, entitled“System and Method for Processing a Video Signal With Reduced Latency,”now U.S. Pat. No. 10,462,381, issued Oct. 29, 2019, which itself is acontinuation-in-part of, and claims priority to, application Ser. No.14/701,075, filed on Apr. 30, 2015, entitled “System and Method forProcessing a Video Signal With Reduced Latency,” now U.S. Pat. No.9,891,438, issued Feb. 13, 2018, the contents of which are incorporatedherein for all purposes.

TECHNICAL FIELD

This disclosure relates to a video magnifier. The present disclosurealso relates to a means of processing a video signal to reduce latency.Also disclosed is a means of video processing whereby shading can beapplied to de-emphasize selected portions of the viewing area within themonitor.

BACKGROUND OF THE INVENTION

Video magnifiers are important tools for individuals who are blind orhave low vision (i.e. “BLV users”). The construction typically includesan arm mounted camera and an associated monitor. The arm allows thecamera to be directly positioned over the object being viewed. Theobject may be, for example, a prescription pill bottle or a newspaper orbook. The arm is usually adjustable to permit the camera to focus upon aparticular portion of the object or document. BLV users can then selecta desired level of magnification so that the object, and any associatedtext, can be sufficiently enlarged for viewing and reading. Colorcontrasting and image rotation can be provided to further ease viewing.An X-Y table can optionally be positioned beneath the camera to allowfor precise movements to the object being viewed. A video processor andlook up tables are used to process the video signal generated by thecamera. The video signal can be processed to make changes to one or moreof the following: image scale; image position; color contrast; as wellas other image features.

For all of these reasons, video magnifiers have become an indispensabletool to members of the BLV community. Nonetheless, traditional videomagnifiers still suffer from some shortcomings. One of theseshortcomings is latency. Latency refers to the slight delay that resultsfrom the video signal being processed. This delay is usually measured inmilliseconds and occurs between the image being captured by the cameraand its subsequent display upon the monitor. The use of frame buffersgenerally increases latency. Frame buffers are memory stores thatcontain a bitmap of an entire frame of data. Frame buffers allow thevideo frame to be scaled and otherwise allows desired attributes to beapplied to the frame. The latency that results from frame buffering islargely unnoticeable in most applications. However, latency is highlyproblematic in the context of a video magnifier, as BLV users willnotice a lag between the manipulation an object and its subsequentappearance on the video monitor. This lag greatly complicates the manualtasks performed by BLV users.

Known video processing systems typically require multiple frame buffers.In a typical video processing system there are three frame buffers, oneto store the incoming frame, one to store the frame that is beingmanipulated in memory, and one to store the output frame. In alow-vision magnification system, there can be many design blocks, whicheach adding more frame buffers. The cumulative total of frame bufferstimes the frame time results in a large delay between an event in frontof the camera and when it appears on the display. This delay or videolatency causes significant problems for low vision users when theyattempt tasks in front of the camera that require eye-hand coordination.Examples include writing during test taking, signing a name to adocument, or attempting to push a needle through fabric while sewing.Every frame buffer adds to the cumulative photon to photon latency,which can be measured on a black box system by triggering a bright pointsource of light in front of the camera system and them measuring thedelta in time to when the light is visible on a monitor.

Video magnifiers also employ techniques to help focus the user's visionupon the screen. For this reason, traditional video magnifiers sometimesblack out regions of the screen. The blacked out regions tend tohighlight the non-blacked out regions. This allows the BLV user to focusupon one area of an object. In the case of text, it allows the BLV userto read one line of text at a time. The underlying text may be scrolledso that sequential lines are presented to the user.

Masks have long been used in security systems to block areas in a scenethat require privacy, such as the windows of an apartment building. Themask's objective is to obliterate or obfuscate the area that should bekept confidential in the scene. In security cameras with PTZ(pan-tilt-zoom), the mask is typically maintained over the protectedarea of the scene as the camera aim is changed. UK Patent Application GB2,472,307 discloses a document camera presentation device with markersplaced on the document to mask selected areas. Although this techniqueis beneficial, it has some drawbacks. By blacking out regions of text,the BLV user often loses their place within the document. It also makesit difficult for the user to navigate around an object and throughlengthy text.

In a low-vision system, the purpose of the mask is to reduce theintegral of light entering the user's eyes. Many low-vision users aresensitive to light and develop eye-strain when using a bright monitor.The masks in a low-vision magnifier are typically placed at the top andbottom of the display output, irrespective of the input scene. Thisleaves a wide central region where magnified items, typically writtenwords, are visible. In the past users would cut up card board boxes andtape them to their monitors to block off a large percent of the displayto help reduce eye-strain from too much light. This method gave rise tothe digital horizontal and vertical masks now used in many low-visionmagnifier systems. While masks solve the problem of too much lightentering the user's eyes they created a new problem, where the user'sfield of view became substantially limited, making the user lose theirspatial perspective in the document or scene they were trying to observeduring eye-hand coordinated tasks.

Various video processing techniques are known in the prior art. Some ofthese techniques discuss reducing latency. For example, WO2013/068584 toEaturnus discloses a ultra-low latency video communication system. Thissystem employs a transmission unit with image acquiring circuitry orimage reconstruction circuitry for acquiring or reconstructing an imageframe or an image field. A video processing unit is also included forprocessing at least part of the video data. Also included is acommunication unit for sending or receiving at least part of the videodata.

Similarly, US Pub 2005/0288932 to Kurzweil discloses a system forreducing processing latency in optical character recognition for aportable reading device. The portable reading device may include acomputing device with a computer readable medium for storing a computerprogram to receive an image and select a portion of the image toprocess. The device then processes the selection of the image with afirst process, and when the first process is finished, a second processis initiated. While the second processing, the first process is repeatedon another section of the image.

US Pub 2013/0329114 to Kim discloses an image magnifier with pointcontrol. The method allows for image control using magnification. Themethod comprises selecting a portion of the image, and then magnifyingthe selected portion. An action is then performed on the magnified andselected portion of the frame. The change may involve changing a featureof the selected portion.

Although these inventions achieve their own unique individualobjectives, all suffer from drawbacks. For example, none of the priorart sufficiently eliminates the problems associated with latency.Likewise, none of the prior art allows shading to be applied to selectedportions of a document in a manner suitable by low vision users.

What is needed, therefore, is a desktop video magnifier camera includesa pass through signal processor that eliminates the use of frame buffersand that greatly reduces associated latency. The video magnifier of thepresent disclosure is designed to fulfill this and other shortcomingspresent with existing video magnifiers.

What is also needed is a magnifier camera that allows portions of anobject to be viewable but shaded. This would allow select portions of anobject or text to be highlighted while at the same time allowing theremainder of the object or text to be perceived. The video magnifier ofthe present disclosure is designed to fulfill this and othershortcomings present with existing video magnifiers.

SUMMARY OF THE INVENTION

This disclosure relates to a video magnifier with low latency.

The disclosed video magnifier has several important advantages. Forexample, by eliminating the frame buffer, any noticeable delay in signalprocessing can be avoided.

Another advantage is achieved by utilizing a pass through videoprocessor that allows users to view objects in near real time, which isespecially important when the objects are being manipulated by BLVusers.

A further advantage is realized by processing the video signal so thatselected portions of the screen are shaded to thereby highlight thenon-shaded portions.

Yet another advantage is realized by allowing certain portions of anobject or text to be highlighted while at the same time allowing theuser to perceive the remaining portions of the document.

Various embodiments of the invention may have none, some, or all ofthese advantages. Other technical advantages of the present inventionwill be readily apparent to one skilled in the art.

BRIEF DESCRIPTION OF THE DRAWINGS

For a more complete understanding of the present disclosure and itsadvantages, reference is now made to the following descriptions, takenin conjunction with the accompanying drawings, in which:

FIG. 1 is a perspective view of the magnifier of the present disclosure.

FIG. 2 is a schematic of the video processor used in connection with thepresent disclosure.

FIG. 3 is a front elevational view showing shading for use by blind orlow vision users.

FIG. 4 illustrates the pixel by pixel signal processing of the presentdisclosure.

FIG. 5 is a flow chart illustrating some of the steps associated withthe present disclosure.

FIG. 6 is an illustration of an additional embodiment of the presentdisclosure.

Similar reference numerals refer to similar parts throughout the severalviews of the drawings.

DETAILED DESCRIPTION OF THE DRAWINGS

This disclosure relates to a video processor for a magnifier camera. Inparticular, the disclosure relates to a video processor that eliminatesthe use of a frame buffer. This, in turn, reduces the latency otherwisepresent in the video signal. The disclosed video processor also allowsselected portions of the display to be shaded. This highlights thenon-shaded portions of the display while at the same time allowing theentire display to be perceived by the user. The various components ofthe present invention, and the manner in which they interrelate, aredescribed in greater detail hereinafter.

FIG. 1 illustrates a desktop video magnifier 10 that is constructedaccordance with the present disclosure. The particular magnifierdepicted is more fully described in co-pending application Ser. No.14/607,197 filed on Jan. 28, 2015 and entitled “Video Magnifier Camerawith Handle.” The contents of this commonly owned application are fullyincorporated herein for all purposes. Video magnifier 10 includes a highdefinition camera 20 that is mounted upon a camera arm 22. Camera arm 22is flexible. Arm 22 can also swivel about a vertical axis and pivotabout one or more horizontal axes to allow the user to properly positioncamera 20 over the object to be viewed. In one embodiment, arm 22 isdimensioned to be received within a trough 24 on the back of the devicehousing 26. Objects to be viewed can be placed directly beneath camera20. Alternatively, an x-y table can be included beneath camera 20 toallow objects to be precisely oriented.

Magnifier 10 is supported by a base 32. Base 32, in turn, supports avideo monitor 34 that is coupled to camera 20 via a cable, such as a VGAor HDMI cable. Signals generated by camera 20 are processed via a videoprocessor 36 and displayed on monitor 34 for viewing by the operator.FIG. 2 illustrates processor 36, which is preferably a FieldProgrammable Gate Array (“FGPA”), and an associated look up table 38(“LUT”). LUT 38 assigns output values based upon the input signal. Inaccordance with the invention, the input signal is processed on apixel-by-pixel basis as opposed to a frame-by-frame basis. Processingthe signal on a pixel-by-pixel basis eliminates the need for a framebuffer. Rather than storage large amounts of data in a memory buffer,processor 36 and LUT 38 can process each individual pixel of therecorded image. This “pass through” processing increases processingtimes and greatly reduces associated latency. The pixel-by-pixelprocessing is diagrammatically illustrated in FIG. 4. In order toaccomplish this, the camera resolution is matched to the monitorresolution.

In the present system the frame buffer is eliminated by making theoutput video synchronous to the input video. As each pixel enters it ismanipulated by low-vision processing algorithm(s), in this case by usinga dynamically programmable LUT, and then the pixel exits as outputvideo. The delay between input pixel and output pixel is only a singlepixel clock, as opposed to the typical three frame delays of othersystems which are millions of pixel clocks.

Processor 36 and LUT 38 can apply any of a number of differentcharacteristics to the displayed image. In one example, the color schemeof the displayed image is altered. For example, images can be displayedin contrasting colors, such as blue/green, red/yellow or black/white.The user may cycle through the various color combinations to find themost effective output.

Alternatively, the video signal can be processed to highlight a desiredregion or regions of the displayed video. This feature of the inventionis depicted in FIG. 3. As illustrated, text is being highlighted on thedisplay by shading upper and lower portions (42 a and 42 b) of the text.This has the desired effect of highlighting the non-shaded portion42(c). The degree of shading can be varied depending upon the needs ofthe particular user. It has been found that such shading is beneficialto BLV users suffering from conditions such as retinitis pigmentosa. Theshading, however, is preferably not so great so as to completely mask orblock out the underling words or images. This prevents the user fromlosing track of their place within the text. It likewise prevents a userfrom losing their orientation in an underlying image. The degree ofshading and the orientation and positioning of the shading can beselected by the user. For example, the shading can be orientated tohighlight a vertical, as opposed to horizontal, portion of the image.

In addition to helping focus upon a particular area, the shading alsoreduces the amount of light entering the user's eyes. This, in turn,reduces eye strain and lessens the pain associated with some low-visiondisabilities. Thus, the semi-transparent shades reduce light from themonitor without limiting the viewable area available to the user.

FIG. 5 is a flow chart illustrating the steps associated with thepresent method of video signal processing. As noted, the method iscarried out using: a camera; a microprocessor with an associated look-uptable; and a monitor. One of the objectives of the method is to processthe video signal in a manner that allows the resulting image to be moreeasily viewed by a blind or low-vision user.

In the first step, the camera is directed toward an object to be viewed.This can be any of a variety of objects the user wishes to examine. Theobject can be placed underneath the camera or the camera can be pointedto a remote object. The camera then generates a video signalcorresponding to the recorded image. This video signal comprises aseries of sequential video frames. In the preferred embodiment, theimage resolution is chosen to match the monitor being employed. Each ofthe individual video frames, in turn, comprises an array of individualpixels or picture elements. The number of pixels is a function of theunderlying image resolution. For example, each frame may include a widthof 1024 pixels and a height of 768 pixels. The video signal is nextdelivered to a video processor.

The signal is then processed by the microprocessor and in accordancewith pre-established instructions from the look-up table. In onepossible, embodiment, the look-up table specifies portions of the arrayto be shaded and the degree of shading to be applied. Preferably thedegree of shading is such that the underlying image is not masked. Bynot completely masking the underlying image, the user is able tomaintain spatial awareness of the object being displayed. As noted inFIG. 3, the shaded portions of the array function to highlight thenon-shaded portions of the array and thereby facilitate viewing by thelow-vision user. The shaded portions can be arranged either verticallyor horizontally depending upon the preferences of the user.

The processed video signal is next delivered to the monitor to bedisplayed. Because the microprocessor processes the video on a pixel bypixel basis the use of an image store, such as a frame buffer, iseliminated. This, in turn, greatly reduces latency.

In an alternative embodiment of the present invention, the systemutilizes a number of different look up tables (38(a), 38(b), and 38(c)).Each of these look up tables contains instructions for onepre-determined visual characteristic to be applied to the underlyingimage. For example, table 38(a) may contain instructions for shadingportions of the array, table 38(b) may contain instructions for variouscolor contrasting to be applied to the image and the background, andtable 38(c) may contain instructions for positioning the object withinthe frame.

The present inventors devised a setup to measure latency. The set upconsisted of a white LED with a switch, a photo detector, and anoscilloscope. Pressing the switch powered the LED, causing a highintensity photon point source to be injected into the camera sensorarray asynchronous to the camera frame rate. Subsequently the photonpoint source appeared on the output display, with the delta betweeninput event and output event measured on the oscilloscope. Manymeasurements were taken to find the minimum and maximum latency. Testswith users determined that a cumulative photon to photon latency of lessthe 100 milliseconds (ms) was required to make low-vision systems usablefor tasks requiring eye-hand coordination. The latency for many deviceson the market was tested and found unsuitable.

The present system overcomes unacceptable latency by eliminating framebuffers in the low-vision enhancement algorithms used with the presentvideo processing systems. The present system also solves the problem oftoo much light and loss of perspective, by using shades with adjustableopacity that can be enabled and sized both horizontally and vertically.This gives the user the ability to observe the area around the centralarea of interest without eye-strain, while adjusting the size andopacity of the shades to their individual comfort. The shades and masksof the present system are inserted in real-time into the pixel streamwithout the use of frame buffers.

Although this disclosure has been described in terms of certainembodiments and generally associated methods, alterations andpermutations of these embodiments and methods will be apparent to thoseskilled in the art. Accordingly, the above description of exampleembodiments does not define or constrain this disclosure. Other changes,substitutions, and alterations are also possible without departing fromthe spirit and scope of this disclosure.

What is claimed is:
 1. A system for processing a video signalcomprising: a desktop video magnifier including a camera, a camera arm,and a video monitor, the camera adapted to view objects and generate acorresponding video signal, the video signal comprising a frame made upof an array of pixels; the camera and the video monitor each having aresolution, with the camera resolution being matched to the videomonitor resolution; a microprocessor and an associated look-up table forprocessing the video signal, the processing applying a visualcharacteristic to the frame, the processing being carried out on a passthrough basis and in the absence of a video buffer, the absence of thevideo buffer eliminating latency present in the video signal; whereinthe monitor displays the video signal with the pre-determined visualcharacteristic to a user.
 2. The system as described in claim 1 whereinthe applied visual characteristic is shading of certain portions of theframe.
 3. The system as described in claim 2 wherein the non-shadedportions of the frame are highlighted.
 4. The system as described inclaim 2 wherein the shading allows the entire frame to be perceived bythe user.
 5. The system as described in claim 1 wherein the appliedvisual characteristic is vertically arranged shading.
 6. The system asdescribed in claim 1 wherein the applied visual characteristic ishorizontally arranged shading.
 7. The system as described in claim 1wherein multiple look-up tables are employed to apply multiple visualcharacteristics to the frame.
 8. A system for processing a video signalcomprising: a camera for viewing an object and generating acorresponding video signal, the video signal comprising a frame made upof an array of pixels; a microprocessor and an associated look-up tablefor processing the video signal, the processing applying apre-determined visual characteristic to the frame, the processing beingcarried out on a pixel by pixel basis, and without a video buffer, theabsence of a video buffer eliminating latency in the video signal; amonitor for displaying the video signal with the pre-determined visualcharacteristic; the camera and the monitor each having a resolution,with the camera resolution being matched to the monitor resolution. 9.The system as described in claim 8 wherein the pre-determined visualcharacteristic is shading that highlights the non-shaded portions of thearray and which thereby facilitates viewing by the low-vision user. 10.The system as described in claim 8 wherein multiple look-up tables areemployed to apply multiple visual characteristics to the frame.